Reduce default checkpoint distance flag from 40 to 20 to speedup EN startup #2223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reduce checkpoint distance flag to trigger creation of checkpoints more frequently. This change will cause fewer WAL segments to trigger checkpoint creation, which will speedup EN startup by having fewer WALs to replay during startup. On benchnet-dev-004, replaying 20 WALs was 3.5 minutes faster than replaying 40 WALs. This change will also speedup checkpoint file creation by the same amount because that also requires replaying WALs.
zhangchiqing
approved these changes
Mar 29, 2022
m4ksio
approved these changes
Mar 29, 2022
| @@ -145,7 +145,7 @@ func main() { | |||
| flags.StringVar(&triedir, "triedir", datadir, "directory to store the execution State") | |||
| flags.StringVar(&executionDataDir, "execution-data-dir", filepath.Join(homedir, ".flow", "execution_data_blobstore"), "directory to use for Execution Data blobstore") | |||
| flags.Uint32Var(&mTrieCacheSize, "mtrie-cache-size", 500, "cache size for MTrie") | |||
| flags.UintVar(&checkpointDistance, "checkpoint-distance", 40, "number of WAL segments between checkpoints") | |||
| flags.UintVar(&checkpointDistance, "checkpoint-distance", 20, "number of WAL segments between checkpoints") | |||
There was a problem hiding this comment.
just maybe we should use some const rather than magic number
There was a problem hiding this comment.
Great point! I'll open an issue/PR to replace magic numbers in the flags. E.g.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reduce checkpoint distance flag to trigger creation of checkpoints more frequently.
Prior to #1944 checkpoint creation took 11-15+ hours so it wasn't feasible to run checkpoints more frequently.
This change will cause fewer WAL segments to trigger checkpoint creation, which will speedup EN startup by having fewer WALs to replay during startup.
On benchnet, replaying 20 WALs is 3.5 minutes faster than replaying 40 WALs.
This change will also speedup checkpoint file creation by the same duration because that also requires replaying WALs.
closes #2207
updates https://github.com/dapperlabs/flow-go/issues/6114